It is common practice for a data storage device, such as a disk-based data storage device, to have a recommended value on the number of requests that it can handle under good conditions and under error conditions. This value is typically referred to as a Queue Depth, and can be used by a disk device driver to control the input/output (I/O) flow to the storage device.
In a multipath configuration environment there are multiple paths to provide I/O with the disk device driver. Since the number of I/O sent from the disk device driver to the storage device is still limited by the Queue Depth value, there can be many jobs that are queued at a disk device driver pending queue. This can cause a problem during error recovery, as the disk device driver will typically retry all the requests on the queue for some number of times (e.g., five retries per queued request). Since the pending queue could become much longer in a multipath configuration environment, this can result in a significant performance degradation during error recovery, or in worst case, the system hanging resulting in an application timeout.
In order to address this problem, a storage multipath device driver can implement Queue Depth control at its level to limit the amount of I/O sent to the disk device driver. This process can aid in solving the performance degradation problem at the level of the disk device driver during an error recovery procedure.
However, a further problem can then be introduced during normal (non-error) conditions with heavy or stress I/O at the storage multipath device driver level, especially with certain types of applications that flood very heavy I/O to a small number of storage devices. Under this condition, a large number of jobs can be enqueued at a pending queue of the storage multipath device driver, which can result in severe performance degradation and/or a system hanging event.
It can be appreciated that absent a Queue Depth limit at the storage multipath device driver level, the disk device driver can become a bottleneck in the error recovery situation. However, if the storage multipath device driver uses Queue Depth to limit I/O flow, then storage multipath device driver can become the bottleneck during normal (non-error) condition with stress I/O.
In US 2004/0,194,095 A1, “Quality of Service Controller and Method for a Data Storage System”, Lumb et al. disclose that requests for each of a plurality of storage system workloads are prioritized. The requests are selectively forwarded to a storage device queue according to their priorities so as to maintain the device queue at a target queue depth. The target queue depth is adjusted in response to a latency value for the requests, where the latency value is computed based on a difference between an arrival time and a completion time of the requests for each workload. Prioritizing the requests can be accomplished by computing a target deadline for a request based on a monitored arrival time of the request and a target latency for its workload. To reduce latencies, it is said that the target queue depth may be reduced when the target latency for a workload is less than its computed latency value, and to increase throughput the target queue depth may be increased when the target latency for each workload is greater than each computed latency value.
In U.S. Pat. No. 6,636,909 B1, “Adaptive Throttling for Fiber Channel Disks”, Kahn et al. disclose a method that sends a write request to a disk and, in response to receiving a queue full signal from the disk if the disk queue is full, sets a throttle value. The method is said to seek to avoid triggering a queue full status for a storage device by queueing commands that would overload the storage device in a local software disk driver queue. Since a predefined limit on command issuance is said to not be feasible, initiator devices instead must be able to recognize potential error producing situations and thereafter limit or throttle the number of commands issued. Accordingly, a method operates by sending a write request to a disk, receiving a queue full signal from the disk if the disk queue is full, and responsive to receiving the queue full signal setting a throttle value and thereafter dynamically adjusting the throttle value to maintain the storage device in a steady state.
In U.S. Pat. No. 6,170,042 B1, “Disc Drive Data Storage System and Method for Dynamically Scheduling Queued Commands”, Gaertner et al. disclose a data storage system and method of scheduling commands in which commands are stored in a command sort queue and a scheduled command queue. Commands in the command sort queue are sorted and assigned a priority. Eventually, commands in the command sort queue are transferred to the scheduled command queue, where commands in the scheduled command queue are executed without further sorting. The desired queue depth or size of the scheduled command queue is determined as a function of both the queue depth of the command sort queue and a command execution rate value indicative of the rate at which commands in the scheduled command queue are executed. The desired queue depth can be dynamically determined using the queue depth of the command sort queue and the command execution rate value as inputs to a look-up table. The data storage system is said may include a small computer system interface (SCSI) disc (or “disk”) drive that executes commands from a host system.
These various U.S. Patents and the U.S. Patent Publication do not address the specific problems discussed above, and thus do not provide a solution for these problems.